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Video Bit Stream Extension with Supplementary Content Information 
TO AID in Subsequent Video Processing 

5 COPYRIGHT NOTICE 

Contained herein is material that is subject to copyright protection. The copyright 
owner has no objection to the facsimile reproduction of the patent disclosure by any 
person as it appears in the Patent and Trademark Office patent files or records, but 
otherwise reserves all rights to the copyright whatsoever. 

10 

BACKGROUND OF THE INVENTION 

(1 ) Field of the Invention 

The present invention is related to the field of video processing, and more 
specifically, to the method and apparatus of video bit stream extension to aid in 
15 subsequent video processing, 

(2) Related Art 

Currently, the motion picture and television industries utilize various video 
processing methods and apparatus to create special effects to captured video. Typically, 
video images are captured by some type of a video capture device, and recorded on a 
20 storage device. One or more video sequences from one or more video sources, such as a 
storage device, may be analyzed to collect information regarding each video sequence. 
The collected information is then utilized to manipulate the video sequence through, for 
example, some type of computer system, to create special effects. For example, video 
sequences from two different sources may be merged by video compositing to create 
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special effects. In order to perform video compositing, information regarding each video 
sequence may be useful 

It is therefore desirable to provide a method and apparatus to generate an extended 
video bit stream for a video sequence. The desired extended video bit stream would 
contain non- visual information useful in several applications including but not limited to 
motion picture and television special effects, post processing, video editing and video 
library management. 
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BRIEF SUMMARY OF THE INVENTION 
A method for annotating video and audio media with supplementary content for 
post video processing is disclosed. The method includes the steps of accepting video data 
from a video source and storing video information associated with the video data as the 
video data is being accepted. Then, the video information may be appended to the video 
data for later use in the form of annotations, for example. 
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BRIEF DESCRIPTION OF THE DRAWINGS 



Figure la is an exemplary system block diagram of a video source and video 
output video processing system according to one embodiment of the present invention. 
5 Figure lb illustrates a video processing information collection and annotation 

system according to one embodiment of the present invention. 

Figure 2 is a block diagram illustrating calibration software configured to 
determine camera geometry information from each frame and/or scene data of a video 
sequence made available through a video source device according to one embodiment of 
10 the present invention. 

Figure 3 is a block diagram of pose estimation software configured to determine 
camera pose information for each frame and/or scene data collected from a video source 
device according to one embodiment of the present invention. 

Figure 4 is a block diagram of illumination input software which is configured to 
15 accept illumination information for a given frame or scene data of a video sequence 
provided through a video source device according to one embodiment of the present 
invention. 

Figures 5a-5c illustrate an annotated video extended bit stream format according 
to one embodiment of the present invention. 
20 Figure 6 is a flow diagram illustrating the general steps followed by annotator 

software configured to annotate a video bit stream with certain video processing 
information according to one embodiment of the present invention. 
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DETAILED DESCRIPTION OF THE INVENTION 



The present invention relates to a method and apparatus for producing an 
annotated video extended bit stream to aid video processing. In one embodiment of the 
present invention, certain video sequence information is collected on a per frame, per 
5 video segment or a per scene basis, as video data is being provided through a video 

source device. Video information is added to a video bit stream to generate an extended 
video bit stream with annotations (hereinafter referred to as an annotated video extended 
bit stream). 

The annotated video extended bit stream of the present invention includes both 
10 the original video sequence as well as certain additional non- visual information. The 
additional non-visual information is useful in subsequent processing of the video data 
provided by a video source device for several applications including special effect 
processing, easy access to video databases, efficient video coding, and easier generation 
of three-dimensional models. 

15 Prior art methods must rediscover the information which the present invention 

annotates by using only the available visual data. Further, the present invention solves 
mathematically ill posed problems for which solutions are not available. Additionally, 
the present invention efficiently provides access to data for which solutions are possible, 
but are obtained through great computational cost. The present invention leads to better 

20 and faster compression via model building than prior art methods which rediscover the 
factors from the captured media only. 

Additionally, the present invention allows better digital manipulation of media. 
For example, three-dimensional models of objects in a scene can be constructed and used 
as search keys in a video database, and geometric and photometric properties can be used 
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as derived search keys. Camera scene content is already often under program control. 
This control information as well as what can be sensed in the camera, such as a 
cameraman turning a zoom lens and camera pose, is conveyed to subsequent processing 
software through the annotated video extended bit stream of the present invention. 

5 Although the present invention is described for use in the motion picture and 

television industries, the present invention may also be usefiil in video cameras used by 
consumers to automatically or manually armotate certain parameters set during recording, 
such as zoom factor or focus. 

Figure la is an exemplary system block diagram illustrating a video source and 
10 video output system according to one embodiment of the present invention. In this 

example, the video processing information collection and aimotation system 104 may be 
implemented for use on a computer 102 with some type of a storage device 103 coupled 
to a processor 105 by bus 107. The computer 102 may be coupled to a video source 100 
configured to provide video data (video sequence) to the computer 102. In one 
15 embodiment, the video source 100 may be, but is not limited to, a video capture device 
106 such as a camera, a video replay device 108 such as a video camera recorder (VCR) 
and a video transmission device 1 1 0 such as a digital or an analog television (TV). 

The present invention's video processing information collection and aimotation 
system 104 collects certain video processing information from video image data (frame or 
20 scene) as they are provided from the video source 100 and generates an annotated video 
extended bit stream comprised of the video image data as well as corresponding video 
processing information. 

Although the video processing information collection and aimotation system 104 
is illustrated as being implemented in a computer 102, a person skilled in the art provided 
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with the detailed description herein may appreciate that the video processing information 
collection and annotation system 104 may also be implemented in the video source 100 
such as a camera. 

Figure lb illustrates a video processing information collection and annotation 
5 system 104 according to one embodiment of the present invention. Video data 200 is 
provided by video source 1 00 in the form of frames (scenes) and forwarded to a collector 
201 . A calibration software 206, such as NCAMERA from REALVIZ of Sophia 
Antipolis Cedex, France, generates camera geometry information 204. The camera 
geometry information 204 may be generated either from the video data 200 provided by 

10 the video source 100 or as the video data is being captured by a video capture device 106. 
The camera geometry information 204 is then forwarded the information on to the 
collector 201 . Similarly, a pose estimation software 210, such as MATCHMOVER from 
REALVIZ of Sophia Antipolis Cedex, France, generates camera pose information 208. 
The camera pose information 208 may be generated from the video data 200 provided by 

15 the video source 100 or at the same time that the video data 200 is being captured by a 
video capture device 106. The camera pose information 208 is then forwarded to the 
collector 201. 

Source identification/description and illumination information is gathered by a 
source identification/description/illumination input software 212 from, for example, input 

20 by a camera operator (user) or a television director operating a television switcher (not 
shown). The information gathering may be fiilly automated by the television switcher. 
The gathered source identification/description/illumination information 21 1 is then 
forwarded along with the camera geometry information 204 and the camera pose 
information 208 to the collector 201. The collector 201 forwards the information 

25 collected to an annotator 202. 
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The current state of the frame data and the gathered information are received and 
compared by a comparator 222. The gathered information and current state are also 
received by a differential generator 230 which determines the differential information. 

More specifically, the auxiliary information is gathered periodically but only 
5 encoded in the bit stream if there are changes. The current state indicates the latest 

encoded information in the bit stream and is kept in memory, e.g., state storage 240. The 
differential information is the difference between the current information and the gathered 
information. If the information remains the same, there is no need to annotate the bit 
stream. 

10 The differential information is encoded by an encoder 224 as auxiliary 

information for the current frame data. The annotator 202 stores the various information 
gathered in the video bit stream 200 by extending the original bit stream format. The 
annotated video extended bit stream 214 may be utilized by a video processor 216 to 
separate video data 218 from auxiliary information 220. The video data 218 is equivalent 

15 to video data 200 before processing by the annotator 202 and the auxiliary information 
220 is additional information gathered regarding the video data 200 including the camera 
geometry information 204, the camera pose information 208 and the source 
identification/description/illumination information 211. 

Figure 2 is a block diagram illustrating the gathering of calibration information 
20 according to one embodiment of the present invention. In this example, camera geometry 
(intrinsic parameters) is provided by a camera calibration procedure which provides a 
camera projection matrix 270. In one embodiment, instrumentation attached to a set of 
actuators 250 is used to control and keep track of the geometry of cameras (e.g., auto- 
focus). Calibration is obtained by using a calibration box 260 of known size and shape 
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with a distinct set of markers on its surface. The calibration software 206 utilizes the 
calibration box 260 to produce the camera projection matrix 270. 

More specifically, in one embodiment, a video capture device 106 such as a 
camera causes a transformation to the incumbent light. This transformation depends on 
5 the physical arrangement of lenses and light sensors of the video capture device that is 
modeled by a 3 x 4 transformation matrix also referred herein as a camera projection 
matrix 270. The twelve parameters from the projection matrix define the camera 
geometry. Various methods for producing the projection matrix 270 are well known. 

Figure 3 is a block diagram illustrating the gathering of camera pose estimation 
10 information according to one embodiment of the present invention. The camera pose 
information 208 includes extrinsic camera parameters. More specifically, the images 
generated by a video capture device such as a camera depends on the position of the video 
capture device 106. The six degrees of freedom of camera pose may be defined with 
respect to a given reference. For example, the position of the camera on the very first 
15 frame may be used as a reference from which the camera pose for the subsequent frames 
may be measured. 

Computation of the pose estimation information 208 may be made by the pose 
estimation software 210. Depth maps are estimated firom a video sequence by using 
structure from motion. For example, the registration of the depth map to a three- 
20 dimensional model of the scene provides the pose information. The three-dimensional 
scene model may be constructed using the registered data in one embodiment. Obtaining 
pose estimation from video data is well known in the art and needs no further discussion. 

Alternatively, camera pose may be provided by a position tracking system based 
on gyroscopes, pendulums or magnetic field sensors (similar to those used in virtual 
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reality helmets) installed in a video capture device 106 and provided to the collector 2d 
by the video capture device 106. Instrumentation attached to a set of actuators 250 
(attached to the video capture device 106) may be used to remotely control the pose of 
robotic cameras remotely. 

5 Figure 4 is a block diagram illustrating the gathering of source 

identification/description and illumination information according to one embodiment of 
the present invention. Video source identification and description is provided by using 
information which is available either at a television (TV) switcher (live production) or 
editing A/B roll machine (off-line production). A TV director/editor defines an editing 
10 list which is implemented by an operator controlling the TV switcher 400 or A/B roll 
machine. 

All cameras and video sources are logged before actual editing. For future 
movies, this process is performed at the story board. Story board drawings and text have 
a one-to-one correspondence to video shoots. Individual shoot description may be 
15 annotated with the final video at production time. 

Video source identification 402, which is part of the source 
identification/description/illumination information 211 delineates one parameter which 
defines a source number. More specifically, production video typically is composed fi'om 
multiple video sources 106 where each clip may be identified at production time. Each 
20 clip has an associated parameter which defines a source number for the clip. 

The video source description 405 of the source 
identification/description/illumination information 211 refers to textual descriptions for 
distinct video sources. For instance, TV directors typically have textual descriptions for 
distinct video sources, such as for a National Basketball Association (NBA) game. 
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"Scoreboard shoot", "court overview", "action on court", "close-up on coach", or "close- 
up of commentator" are different source descriptions. 

Finally, the illumination condition 403, also part of the source 
identification/description/illumination information 211, refers to computer controlled sets 
5 and lighting system 404. The controls may be armotated as part of the scene description 
as well. For insertion of computer-generated actors in a scene, lighting information is 
fundamental on a per-frame basis. 

Figures 5a - 5c illustrate an exemplary annotated video extended bit stream 
format according to one embodiment of the present invention. The exemplary annotated 

10 video extended bit stream 500 illustrated in Figure 5a has ancillary information 502 
which is a list of parameters of arbitrary length. The parameters are described in more 
detail in a corresponding entry in a tag lookup table 503 illustrated in Figure 5c, hi one 
embodiment, the tag lookup table 503 resides in storage device 103. The parameters 
having the information described in Figures 2, 3 and 4 are valid until a new setting 

15 appears in the annotated video extended bit stream 500. 

Figure 6 is a flow diagram illustrating the general steps followed by annotator 
software configured to annotate a video bit stream with certain video processing 
information according to one embodiment of the present invention. In step 601, frame 
data is obtained from a video source. In step 602, information regarding the frame data, 
20 including camera geometry 204, camera pose 208 and source identification description 
illumination software 21 1 is gathered. In step 603, gathered information is stored in a 
state storage device. In step 604, the current state of the frame data and the gathered 
information is compared by a comparator 222. 
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In step 605, differential information is generated according to the results of the 
comparison. More specifically, the auxiliary information is gathered periodically but only 
encoded in the bit stream if there are changes. The current state indicates the latest 
encoded information in the bit stream and is kept in memory. The differential 
5 information is the difference between the current information and the gathered 

information. If the information remains the same, there is no need to annotate the bit 
stream. 

In step 606, the differential information is encoded by an encoder 224 as auxiliary 
information for the current frame data. In step 607, the auxiliary information is annotated 
10 to the encoded frame data to generate an annotated video extended bit stream 214. 

What has been described is a method and apparatus for annotating video and 
audio media with supplementary content for subsequent video processing. Prior art 
methods must rediscover the information which the present invention annotates by using 
only the available visual data. The present invention solves mathematically ill posed 
15 problems for which solutions are not available. Additionally, the present invention 
efficiently provides access to data for which solutions are possible, but are obtained 
through great computational cost. The present invention leads to better and faster 
compression via model building than prior art methods which rediscover the factors from 
the captured media only. 

20 While certain exemplary embodiments have been described in detail and shown in 

the accompanying drawings, it is to be understood that such embodiments are merely 
illustrative of and not restrictive on the broad invention, and that this invention is not to 
be limited to the specific arrangements and constructions shown and described, since 
various other modifications may occur to those with ordinary skill in the art. 
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What is claimed is: 



CLAIMS 



1 LA method of processing video data comprising the steps of: 

2 accepting video data from a video source; and 

3 storing video information associated with the video data as the video data is being 

4 accepted, the video information being stored as annotations to the video data. 

1 2. The method of claim 1 , wherein the video information comprises camera 

2 geometry information. 

1 3. The method of claim 1 , wherein the video information comprises camera 

2 pose information. 

1 4. The method of claim 1 , wherein the video information comprises source 

2 identification/description/illumination information. 

1 5. The method of claim 1, wherein the video data comprises frame data 

2 obtained from an encoder. 
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1 6 The method of claim 1 , wherein the video source comprises a video 

2 capture device. 



1 7. The method of claim 5, further comprising gathering a current state of the 

2 frame data. 

1 8. The method of claim 7, further comprising comparing the current state 

2 with the camera geometry information, the camera pose information and the source 

3 identification/description/illumination information of the frame data. 

1 9. The method of claim 8, fiirther comprising determining differential 

2 information from a result of the comparing step. 

1 10. The method of claim 9, further comprising appending the differential 

2 information to the frame data. 

1 1 L An apparatus comprising: 

2 a collector configured to collect video information to be associated with video 

3 data gathered from a video source; and 

4 an annotator coupled to the collector, the annotator configured to aimotate the 

5 video data with the video information. 
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12, The apparatus of claim 1 1 , wherein the video information comprises one 
or more of: camera geometry; camera pose information; and source 
identification/description/illumination information. 



1 13. The apparatus of claim 12, further comprising calibration software 

2 configured to generate the camera geometry information for the video data as the video 

3 data is being gathered by the video source and provide the generated camera geometry 

4 information to the collector. 

1 14. The apparatus of claim 12, further comprising pose estimation software 

2 and configured to generate the camera pose information for the video data as the video 

3 data is being gathered by the video source and provide the generated camera pose 

4 information to the collector. 

1 15. The apparatus of claim 12, fiirther comprising an encoder configured to 

2 encode the video data and generate frame data as an input to the annotator. 

1 16. The apparatus of claim 15, wherein the encoder forwards a current state of 

2 the frame data to a state storage device coupled to the encoder. 
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1 1 7. The apparatus of claim 1 6, flirther comprising a comparator configured to 

2 compare the current state with the video information of the frame data. 



1 18. The apparatus of claim 1 7, further comprising a differential information 

2 generator coupled to the state storage device, the comparator and the encoder and 

3 configured to generate differential information from the result of the comparison 

4 performed by the comparator and forward the differential information to the encoder. 



1 1 9. A computer readable media containing executable computer program 

2 instructions which when executed on a digital processing system causes the system to 

3 perform a method comprising: 

4 accepting video data from a video source; and 

5 storing video information for the video data as the video data is accepted, the 

6 video information being stored as annotation to the video data for later use. 

1 20. The computer readable media of claim 19, wherein the video information 

2 is camera geometry information. 

1 21. The computer readable media of claim 1 9, wherein the video information 

2 is camera pose information. 
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1 22, The computer readable media of claim 19, wherein the video information 

2 is source identification/description/illumination information. 



1 23 . The computer readable media of claim 1 9, wherein the video data is a 

2 frame data obtained from an encoder, 

1 24. The computer readable media of claim 19, wherein the video source is a 

2 video capture device. 

1 25 . The computer readable media of claim 1 9, further comprising gathering 

2 current state of the frame data. 

1 26. The computer readable media of claim 25, further comprising comparing 

2 the current state with the camera geometry information, the camera pose information and 

3 the source information of the frame data. 



1 27, The computer readable media of claim 26, further comprising determining 

2 differential information from result of the comparing. 
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1 28. The computer readable media of claim 27, further comprising appending 

2 the differential information to the frame data. 
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ABSTRACT 

A method and apparatus are provided for annotating video and audio media with 
supplementary content for post video processing. The method includes the steps of 
accepting video data from a video source and storing video information associated with 
the video data as the video data is being accepted. Then, the video information may be 
appended to the video data for later use in the form of annotations, for example. 
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I hereby claim the benefit under title 35, United States Code, Section 1 19(e) of any United States 
provisional application(s) listed below 



(Application Number) 



Filing Date 



(Application Number) 



Filing Date 



I hereby claim the benefit under Title 35, United States Code, Section 120 of any United States 
application(s) listed below and, insofar as the subject matter of each of the claims of this application 
is not disclosed in the prior United States application in the manner provided by the first paragraph 
of Title 35, United States Code, Section 112, 1 acknowledge the duty to disclose all information 
known to me to be material to patentability as defined in Title 37, Code of Federal Regulations, 
Section 1.56 which became available between the filing date of the prior application and the national 
or PCT international filing date of this application: 



(Application Number) Filing Date (Status - patented, 



I hereby appoint Farzad E. Amini, Reg. No. P42,261 ; Aloyslus T. C. AuYeung, Reg. No. 35,432; Amy M. 
Armstrong, Reg. No. 42,265; William Thomas Babbitt, Reg. No. 39,591; Carol F. Barry, Reg. No. 41,600; 
Jordan Michael Becker, Reg. No. 39,602; Bradley J. Bereznak, Reg. No. 33,474; Michael A. Bernadicou, 
Reg. No. 35,934; Roger W. Blakely, Jr., Reg. No. 25,831; Gregory D. Caldwell, Reg. No. 39,926; Kent M. 
Chen, Reg. No. 39,630; Lawrence M. Cho, Reg. No. 39,942; Yong S. Choi, Reg. No. P43,324; Thomas M. 
Coester, Reg. No. 39,637; Roland B. Cortes, Reg. No. 39,152; Barbara Bokanov Courtney, Reg. No. 
42,442; Michael Anthony DeSanctis, Reg. No. 39,957; Daniel M. De Vos, Reg. No. 37,813; Robert Andrew 
Diehl, Reg. No. 40,992; Tarek N. Fahml, Reg. No. 41,402; James Y. Go, Reg. No. 40,621; Richard Leon 
Gregory, Jr., Reg. No. 42,607; Dinu Gruia, Reg. No. P42,996; David R. Halvorson, Reg. No. 33,395; 
Thomas A. Hassing, Reg. No. 36,159; Phuong-Quan Hoang, Reg. No. 41,839; Wlllmore F. Holbrow III, 
Reg. No. P41,845; George W Hoover II, Reg. No. 32,992; Eric S. Hyman, Reg. No. 30,139; Dag H. 
Johansen, Reg. No. 36,172; William W. Kidd, Reg. No. 31,772; Michael J. Mallie, Reg. No. 36,591; Andre 
L. Marais, under 37 C.F.R. § 10.9(b); Paul A. Mendonsa, Reg. No. 42,879; Darren J. Milliken, Reg. 
42,004; Thinh V. Nguyen, Reg. No. 42,034; KImberley G. Nobles, Reg. No. 38,255; Michael A. Proksch, 
Reg. No. 43,021; Babak Redjaian, Reg. No. 42,096; James H. Salter, Reg. No. 35,668; William W. 
Schaal, Reg. No. 39,018; James C. Scheller, Reg. No. 31,195; Anand Sethuraman, Reg. No. P43,351; 
Charles E. Shemwell, Reg. No. 40,171; Maria McCormack Sobrino, Reg. No, 31,639; Stanley W. Sokoloff, 
Reg. No. 25,128; Allan T. Sponseller, Reg. No. 38,318; Judith A. Szepesi, Reg. No. 39,393; Vincent P. 
Tassinari, Reg. No. 42,179; Edwin H. Taylor, Reg. No. 25,129; George G. C. Tseng, Reg. No. 41,355; 
Lester J. Vincent, Reg. No. 31,460; John Patrick Ward, Reg. No. 40,216; Stephen Warhola, Reg. No. 
43,237; Charles T. J. Welgell, Reg. No. 43,398; Ben J. Yorks, Reg. No. 33,609; and Norman Zafman, 
Reg. No. 26,250; my attorneys, and James A. Henry, Reg. No, 41,064; Daniel E. Ovanezian, Reg. No. 
41,236; Glenn E. Von Tersch, Reg, No. 41,364; and Chad R. Walsh, Reg. No. 43,235; my patent agents, 
of BLAKELY, SOKOLOFF, TAYLOR & ZAFMAN LLP, with offices located at 12400 Wilshire Boulevard. 
7th Floor, Los Angeles, California 90025, telephone (310) 207-3800, and James R. Thein, Reg. No. 
31 ,71 0, my patent attorney; with full power of substitution and revocation, to prosecute this application and 
to transact all business in the Patent and Trademark Office connected herewith. 



Send correspondence to Michael Anthony DeSanctis BLAKELY, SOKOLOFF, TAYLOR & 
ZAFMAN LLP, 12400 Wilshire Boulevard 7th Floor, Los Angeles, California 90025 and direct 
telephone calls to Michael Anthony DeSanctis (303) 740-1980, 



pending, abandoned) 



(Application Number) 



Filing Date 



(Status ~ patented. 



pending, abandoned) 
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I hereby declare that all statements made herein of my own knowledge are true and that all 
statements made on Information and belief are believed to be true; and further that these 
statements were made with the knowledge that willful false statements and the like so made are 
punishable by fine or imprisonment, or both, under Section 1001 of Title 18 of the United States 
Code and that such willful false statements may jeopardize the validity of the application or any 
patent issued thereon. 



it Invantor Christopher J. Lord ^_ 

^%^^0.AA Date r/>y^1 



Full Name of Sole/First Invi^tor Christopher J. Lord 
Inventor's Signature 



Residence Portland, Oregon Citizenship United States of America 

(City, State) (Country) 

Post Office Address 9170 North West Stark Court 

Portland, Oregon 97229 



Full Name of Second/JoipUnvemod Fernandb C. M. Martins 




Inventor's Signature^ h^^J^^ Date Qg/jgS/^^ 

Residence Hillsboro J Oregon [ Citizenship Brazil 



(City, State) (Country) 



Post Office Address 330-B North East Autumn Rose Way 



Hillsboro. Oregon 97124 



: Inventor Briar>^. Nickerson 

Date 



Full Name of Third/Joint Inventor Brians. Nickerson 
Inventor's Signature 



Residence Aloha, Oregon Citizenship United States of America 

(City, State) (Country) 



Post Office Address 9385 S.W. 192"^ Place 



Aloha. Oregon 97007 
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Title 37, Code of Federal Regulations, Section 1.56 
Duty to Disclose Information Material to Patentability 



(a) A patent by its very nature is affected with a public interest. The public interest Is best served, 
and the most effective patent examination occurs when, at the time an application Is being examined, the 
Office is aware of and evaluates the teachings of all information material to patentability. Each individual 
associated with the filing and prosecution of a patent application has a duty of candor and good faith in 
dealing with the Office, which includes a duty to disclose to the Office all information known to that individual 
to be material to patentability as defined in this section. The duty to disclosure Information exists with respect 
to each pending claim until the claim is cancelled or withdrawn from consideration, or the application becomes 
abandoned. Information material to the patentability of a claim that is cancelled or withdrawn from 
consideration need not be submitted if the information is not material to the patentability of any claim 
remaining under consideration in the application. There is no duty to submit information which is not material 
to the patentability of any existing claim. The duty to disclosure all information known to be material to 
patentability is deemed to be satisfied if all information known to be material to patentability of any claim 
issued in a patent was cited by the Office or submitted to the Office in the manner prescribed by §§1 .97(b)-(d) 
and 1 .98. However, no patent will be granted on an application in connection with which fraud on the Office 
was practiced or attempted or the duty of disclosure was violated through bad faith or intentional misconduct. 
The Office encourages applicants to carefully examine: 

(1) Prior art cited in search reports of a foreign patent office in a counterpart application, and 

(2) The closest information over which individuals associated with the filing or prosecution of a 
patent application believe any pending claim patentably defines, to make sure that any material information 
contained therein is disclosed to the Office. 

(b) Under this section, information is material to patentability when it is not cumulative to 
information already of record or being made or record in the application, and 

(1) It establishes, by itself or in combination with other information, a prima facie case of 
unpatentability of a claim; or 

(2) It refutes, or Is inconsistent with, a position the applicant takes in: 

(i) Opposing an argument of unpatentability relied on by the Office, or 

(ii) Asserting an argument of patentability. 

A prima facie case of unpatentability is established when the information compels a conclusion that a claim Is 
unpatentable under the preponderance of evidence, burden-of-proof standard, giving each term in the claim 
its broadest reasonable construction consistent with the specification, and before any consideration is given to 
evidence which may be submitted in an attempt to establish a contrary conclusion of patentability. 

(c) Individuals associated with the filing or prosecution of a patent application within the 
meaning of this section are: 

(1) Each inventor named in the application; 

(2) Each attorney or agent who prepares or prosecutes the application; and 

(3) Every other person who is substantively involved in the preparation or prosecution of the 
application and who is associated with the inventor, with the assignee or with anyone to whom there is an 
obligation to assign the application. 

(d) Individuals other than the attorney, agent or inventor may comply with this section by 
disclosing information to the attorney, agent, or inventor. 
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